1. SQL queries - This has been somewhat doable for years with NotesSQL, and now with NSFDB2. It needs to become a core part of the product. It could, with more effort from IBM. With the move to Javascript as a primary language, I wouldn't be shocked if IBM implemented SQL as a major query language.
2. JOIN capabilities - This is partially addressed with #1, and you can write backend code or use XPages in 8.5 to do the UI-equivalent but again, indexing speed is key. There is currently NO way to *store* the index that results of a JOIN -- at least not without NSFDB2 and again, it needs to become a core part of the product. The lack of JOIN capabilities is really the main architectural disadvantage of NSF (besides just getting big as we've discussed). There has been a *tiny* bit of progress here in 8.0 with the new NotesDocumentCollection.Intersect() methods and few others. This might suddenly become possible if view index storage and contention was relieved by moving indexes outside of the NSF (commented on my blog).
3. Enforcing Referential integrity - If a doc is "dependent" on another doc, not much is keeping you from deleting the 2nd doc, therefore causing an invalid relationship in the database. This would likely be a lot of work, and when taking replication and doc security into account I'm not even sure it's completely doable. But for many applications this is a "nice to have" that can be addressed in other ways and not an absolute requirement at the db level.
4. Transactional model - IMHO, NSF can't really go there while maintaining it's current capabilities, and this would be the #1 reason to stick with a relational database. People don't enjoy replication/save conflicts in their payment systems.
If you really, really needed this, you could code your own transaction-esque model, too.